{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "13d4c337",
   "metadata": {},
   "source": [
    "These pip installs need to be adapted to use the appropriate release level. Alternatively, The venv running the jupyter lab could be pre-configured with a requirement file that includes the right release. Example for transform developers working from git clone:\n",
    "\n",
    "    make venv \n",
    "    source venv/bin/activate \n",
    "    pip install jupyterlab\n",
    "    venv/bin/jupyter lab"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "9ccb4e2f",
   "metadata": {},
   "outputs": [],
   "source": [
    "%%capture\n",
    "## This is here as a reference only\n",
    "# Users and application developers must use the right tag for the latest from pypi\n",
    "%pip install data-prep-toolkit\n",
    "%pip install data-prep-toolkit-transforms[code2parquet]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "5310a9d7",
   "metadata": {},
   "outputs": [],
   "source": [
    "from dpk_code2parquet import Code2Parquet"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "987876ec",
   "metadata": {},
   "source": [
    "** For a full list of configuration and command line arguements, please refer to [README](README.md)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f194bdb9-cbc4-4435-9639-2559bc39cbdc",
   "metadata": {},
   "outputs": [],
   "source": [
    "Code2Parquet(\n",
    "  input_folder=\"./test-data/input/\",\n",
    "  output_folder=\"output\",\n",
    "  data_files_to_use = ['.zip',],\n",
    "  code2parquet_supported_langs_file = \"./test-data/languages/lang_extensions.json\",\n",
    "  code2parquet_detect_programming_lang = True\n",
    ").transform()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "72581e48-ab1e-46fa-9ebb-a7864001fa0e",
   "metadata": {},
   "source": [
    "***** Use ray runtime to invoke the transform\n",
    "**** The specified folder will include the transformed parquet files."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "24ace3c7",
   "metadata": {},
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "22afe1c0-1c16-4176-9c79-871a86cf88fa",
   "metadata": {},
   "outputs": [],
   "source": [
    "import glob\n",
    "glob.glob(\"output/*\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "31f71d28-e199-41f2-8210-6e14a5db0b1c",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.10"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}
